fix(memory): improve child retrieval and document BEAM v2 - #29

Closes #27

Summary

This PR improves OpenContext's core raw-message retrieval strategy and documents the resulting BEAM 128k evaluation.

The changes are divided into two main parts:

1. Core retrieval and context optimization
2. BEAM v1/v2 diagnostic reports

Retrieval optimizations

• Preserve each complete RawMessage as the parent record.
• Split oversized messages into independently embedded search children using a 400-token target and 80-token overlap.
• Restore either the complete short parent or a continuous matched-child window for long messages.
• Generate semantic and lexical candidates independently and merge them with RRF.
• Overfetch four times the requested limit, deduplicate by parent message, rerank the candidate window, and apply Top-K only after reranking.
• Add an optional local Transformers.js cross-encoder reranker.
• Expose backend, channel, candidate, reranker, and semantic-degradation diagnostics.
• Use the same child indexing and retrieval semantics across SQLite, Chroma, LanceDB, and Milvus adapters.
• Keep the BEAM adapter limited to one upstream turn per RawMessage without benchmark-specific chunking or ranking rules.
• Preserve the official BEAM Answerer, Judge, prompt, and nugget scoring behavior.

Reports

• Translate the original BEAM 128k v1 diagnostic report into English.
• Add a new v2 report covering:
  • parent and child index construction;
  • retrieval and reranking evidence;
  • token and context reduction;
  • category and failure-stage analysis;
  • comparison with the v1 baseline;
  • remaining limitations and recommended follow-up work.

BEAM 128k result

The v2 run completed all 400 questions with zero final execution errors. Semantic and lexical candidates were visible for all questions, and the local reranker changed the candidate order for all 400 retrievals.

Remaining issues

• Multi-evidence retrieval remains the main bottleneck: 57 retrieval misses and 38 partial retrievals remain.
• Only 41.41% of retrieval-applicable questions received every required source.
• Event ordering, summarization, and abstention remain weaker categories.
• The current pointwise reranker improves relevance order but does not optimize evidence-set coverage.
• Local reranking adds approximately 2.85 seconds per question.
• The complete BEAM run covered the SQLite path only; Chroma, LanceDB, and Milvus were not evaluated against live external services.
• The v2 result was produced from an uncommitted working tree, so a future formal baseline should record the dirty diff or run from a clean commit.

Validation

• @melandlabs/ai-rag: 10 tests passed
• @melandlabs/memory-store: 176 tests passed
• @melandlabs/shared: 73 tests passed
• @melandlabs/sqlite: 23 tests passed
• @melandlabs/rag: 30 tests passed
• @melandlabs/indexeddb: 10 tests passed
• @melandlabs/okf: 140 tests passed
• BEAM: 6 tests passed
• TypeScript typecheck passed for shared, sqlite, rag, indexeddb, okf, memory-store, opencontext, and BEAM
• git diff --check passed

No paid benchmark or external backend service is exercised by the test suite.